Australian Journal of Adult Learning 
Volume 50, Number 1, April 2010 


Becoming an Australian citizen: Some dimensions 
of assessing a citizenship-type literacy amongst 

adults 

James A Athanasou 
Adjunct Associate Professor 
La Trobe University 


This paper evaluates a 20-item assessment of citizenship literacy 
in an adult sample comprising 179 persons of English-speaking 
and non-English speaking background. The results indicated that 
the assessment was internally consistent and that as expected 
it distinguished English-speaking from non-English speaking 
participants. The pattern of answers provided an initial, albeit 
partial, understanding of what might constitute citizenship 
information but it also highlighted some limitations. The assessment 
failed to tap the ability levels of those with higher knowledge. 
Nevertheless, the results also indicated some deficits in adult general 
knowledge. The potential failure rate even with a cut-off point of 
60% correct was quite high. Just over one-in-five failed to pass. The 
results have implications for the proposed revisions to the Australian 
Citizenship Test. 
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In December 2006, the Australian Government announced a 
controversial intention to introduce a citizenship test. This followed 
the example of other citizenship testing programs in the United 
Kingdom, the United States and Canada. Eventually the Australian 
Citizenship Amendment (Citizenship Testing) Bill 2007 was 
introduced into Parliament in May 2007; it was passed by Parliament 
in September 2007 and the citizenship test was implemented in 
October 2007. 

The Australian Citizenship Amendment (Citizenship Testing) Act 
2007 No. 142, 2007 stated inter alia that a person is eligible to 
become an Australian citizen if the Minister is satisfied that the 
person: 

(e) possesses a basic knowledge of the English language; 
and (f) has an adequate knowledge of Australia and of the 
responsibilities and privileges of Australian citizenship; and ... 
(e) and (f) are taken to be satisfied if and only if the Minister is 
satisfied that the person has, before making the application: (a) 
sat a test approved in a determination under section 23A; and 
(b) successfully completed that test (worked out in accordance 
with that determination). 

The actual citizenship test is described in official documents as a 
computer-based, written, multiple-choice test. The Standard Test is 
limited to 45 minutes in duration. It is stated that the test is designed: 

... to assess whether you have a basic knowledge of the English 
language. It is also used to test your knowledge of Australia 
and the responsibilities and privileges of citizenship. The test 
consists of 20 questions drawn at random from a pool of 200 
questions. To pass the test, you must correctly answer 60 per 
cent of the questions, including answering three questions on 
the responsibilities and privileges of citizenship correctly... The 
test includes questions about: Australia’s history; Australia’s 
geography; the Australian people; Australian values; the system 
of government; responsibilities and privileges of Australian 
citizenship (Australian Government 2007: 43). 
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This reflects the longstanding legislative requirement for an applicant 
to have an ‘adequate knowledge of Australia’ in the Nationality and 
Citizenship Act 1948 (see i2(i)(e)). There has also been a requirement 
for English in the same Act (see I2(i)(d)). 

Between 1 October 2007 and 31 March 2009,111,005 clients sat the 
Australian citizenship test with 96.7 per cent passing the test on 
their first or subsequent attempt. On average, there were 1.2 tests 
administered per client (Department of Immigration and Citizenship 
2009: 4). 

An independent review of the Australian citizenship test was 
undertaken in 2008 (Australian Citizenship Test Review Committee 
2008). The Citizenship Test Review Committee recommended a 
number of changes. The key changes that will be implemented are: (a) 
the test questions will be rewritten in plain English; (b) the test will 
not contain any mandatory questions; and (c) the current pass mark 
will increase from 60 per cent to 75 per cent. The new citizenship test 
was planned to begin in late September 2009. The intention was that 
the test will continue to assess whether clients possess a basic level of 
English, that is, having ‘a sufficient knowledge of English to be able to 
exist independently in the wider Australian community’ (Australian 
Citizenship Test Review Committee 2008: 5). 

At first glance, the topic of an Australian citizenship test might seem 
to be parochial, practical, political and hardly of theoretical interest 
to adult learning. The issue, however, does have importance for the 
following reasons: (a) a concern about what constitutes citizenship, 
especially the structure of citizenship information; (b) the role of 
external factors such as language, years of residence, education, 
country of origin or general information; and (c) the psychometric 
properties of such an assessment and the relative difficulty of 
particular questions. The purpose of this paper is to evaluate some 
aspects of a citizenship-type assessment that is consistent with the 
content specified by the Australian Government. While the political 
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rationale and the practical implications of citizenship testing are well 
beyond the scope of the paper, the technical aspects of the assessment 
of this construct are of direct relevance to adult learning. 

In the first instance, the Australian Government has made it quite 
explicit that the test assesses not only citizenship but also ‘literacy 
skills’. In this sense, the assessment might be expected to distinguish 
between those persons whose standard of English is adequate to 
undertake the test and those who are not able to read the questions 
(for the most part, this would be those persons whose background is 
a language other than English). Secondly, little is known about the 
nature of this hypothetical latent characteristic that has been called 
citizenship. Aside from the proposed content of the assessment, there 
is no formal understanding of what might constitute a dimension of 
citizenship. That is, in thinking of citizenship knowledge, it might be 
helpful to know what aspects are easiest to acquire and which aspects 
are more difficult. Thirdly, although citizenship has been proposed 
as a desirable quality, there is absolutely no information about how it 
is distributed in the general population. Finally, there is no sensible 
starting point for an individual researcher to come to grips with the 
dimension of citizenship or its assessment because the details of the 
assessment are not public. This situation is unlike that of the United 
States, for instance, where the pool of questions are known in advance 
and the items are selected from that pool. 

In the publicity preceding the Australian Citizenship test the 
Government, however, made available sample questions to the 
press to provide an indication of the likely content. These sample 
questions are the subject of this paper. These were administered to 
a heterogeneous group and the results analysed using psychometric 
(Rasch) item analysis. This report provides a detailed description of 
the citizenship dimension and its constituent parts. The analysis was 
conducted for an entire group as well as for those of English-speaking 
background and those of a non-English speaking background. 
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Methodology 

Participants 

The sample comprised 179 respondents recruited by adult education 
students as a sample of convenience from TAFE classes, AMES 
groups, friends and relatives. No claim is made that the sample is 
representative. It comprised 81 from an English-speaking background 
and 98 from a non-English speaking background. No additional 
demographic details were collected as anonymity and confidentiality 
were key aspects of the data collection because in some groups 
(such as small TAFE or language classes) people might be able to be 
identified if they indicated their age, sex and English or non-English- 
speaking background. 

Instrument 

The instrument comprised 20 multiple-choice questions that were 
publicised as indicative of the types of questions that would be asked. 
These questions are listed in Appendix A. It is recognised that these 
questions may or may not form part of the ultimate test, which in 
any case comprises 20 questions from an intended pool of some 200 
questions. The Cronbach alpha internal consistency coefficient of this 
specific test was high (.836, N = 140) and indicated that the responses 
were homogeneous and the assessment was internally consistent. Of 
course, this is only a partial indicator of the potential reliability of the 
assessment results. 

Procedure 

Participants were asked to participate voluntarily in a study of 
citizenship knowledge and were provided with a sample question and 
instructions. They were provided with their results. Participation was 
voluntary and not tied to any course requirement. A few participants 
declined to participate but no records were maintained. 
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Analysis 

The results were analysed using a traditional item analysis and also 
using Rasch analysis which allows both the difficulty of the items and 
the citizenship knowledge of the participants to be plotted on the 
same dimension. 

Results 

Overall results 

The distribution of results from the combined sample showed that 
most people would have passed the 60 per cent (12 correct out of 20) 
required for citizenship (see Figure 1). On closer inspection and when 
dividing the sample into two groups—English-speaking background 
and non-English speaking background—there are some major 
differences in the pattern of scores. This is illustrated in Figure 2. As 
expected the overwhelming majority of the participants who were of 
English-speaking background (median score = 17) would have passed 
but, in the case of the non-English speaking background, participants 
the pattern of results was entirely different. Despite the fact that the 
majority of non-English speaking background participants would 
have passed, the results (median score = 13) were spread across the 
entire range. 
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Figure 1: Distribution of total scores on the citizenship assessment 
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The average score for English-speaking participants was 
16.2 (SD= 3.5) whereas for non-English-speaking participants the 
average was 12.8 (SD= 4.0). For such an assessment to be valid it 
would be expected to differentiate between groups. The difference in 
scores of the two groups in this instance was statistically significant 
(t(i35)=-5-i8, pco.oooi) and supports the validity of the assessment, 
irrespective of one’s views about the propriety of such assessments. 
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Figure 2: Distribution of total scores on the citizenship assessment 
for Non-English-Speaking Background (NESB) and 
English-Speaking Background (ESB) participants 

Analysis of individual questions 

There was a wide variation in the range of difficulty of the questions. 
The most difficult question for the group was: ‘Who is Australia’s 
head of state?’ (answered correctly by 28%) and the easiest was: 
‘Where did the first European settlers to Australia come from?’ 
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(answered correctly by 95%). The item difficulties are shown in 
column 2 of Table 1 (this indicates the proportion answering the 
question correctly). The third column indicates the relationship of 
each question by itself to the overall score. All questions correlated 
positively with the overall or total score, indicating that the questions 
were in line with the overall assessment results. The questions with 
the lowest item-total correlation appeared to be those that tapped the 
most or least familiar concepts and facts. 

Table 1: Item statistics for the citizenship assessment 


Items 

Proportion 

who 

answered 

correctly 

Item-total 

point 

biserial 

correlation 

1. 

Which colours are represented on the 
Australian flag? 

.90 

•341 

2. 

Indigenous people have lived in Australia 
for... 

•57 

•567 

3- 

Australia’s national flower is the ... 

.60 

•493 

4- 

Which is a popular sport in Australia? 

.90 

.492 

5- 

Australia’s political system is a ... 

•89 

.424 

6. 

The Capital of Australia is... 

•93 

•431 

7- 

Which animals are on the Australian 

Coat of Arms? 

.86 

•459 

8. 

Where did the first European settlers to 
Australia come from? 

•95 

•394 

9- 

Who is Australia’s head of state? 

.28 

•303 

10. 

Who was the first Prime Minister of 
Australia? 

•63 

■571 

11. 

What song is Australia’s national 
anthem? 

•85 

.480 

12. 

What do you call the elected head of a 
state government? 

.66 

•594 
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Items 

Proportion 

who 

answered 

correctly 

Item-total 

point 

biserial 

correlation 

13 . 

Which federal political party or parties are in 
power? 

•73 

.617 

14 - 

Which of the following are Australian values? 

•74 

.489 

15. 

Australia’s values are based on the ... 

•52 

•514 

16. 

What does Anzac Day commemorate? 

•65 

.690 

17 - 

In what year did the first European settlers 
arrive? 

•50 

.496 

18. 

How many states are there in Australia? 

•55 

•329 

19 - 

Australian soldiers fought in ... 

•57 

.619 

20. 

What is Australia’s biggest river system? 

.80 

•539 


Item Response Analysis 

A further analysis using the Rasch model was undertaken to describe 
the link between the construct that was being assessed and how the 
various items were located along the dimension of that construct. 

(The detailed statistics and output form the Rasch analysis are 
available freely from the author upon request. Separate analyses are 
also available for the English-speaking and non-English speaking 
samples.) 

The item map (Figure 3) represents the ability of the group (that 
is, citizenship literacy) and the difficulty of the item. The left-hand 
side of the figure represents a chart of the ability scores of the group. 
It is similar to Figure 1. Each X on the chart represents around 
two persons. If the chart were turned on its side, it would be like a 
histogram of scores. The right-hand of the chart shows the location of 
each of the 20 items on the scale of citizenship ability. The items are 
represented in shorthand fashion as It for Item 1 and I10 for Item 10 
and so on. The scale however might be unfamiliar to some readers. 

It varies from +5 to -5 with zero being the average level of ability and 
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also the average level of difficulty. Persons and items closest to +5 
have the highest ability and are the most difficult, respectively. Items 
near -5 are the easiest and persons closest to -5 are those who are 
least in citizenship literacy. This is an arbitrary scale but one that is 
well-known. The importance of the scale is that someone who is at a 
particular ability level can be expected to answer all the items below 
his/her ability level but have difficulty with the items above his/her 
ability. So a person at around average or zero on the scale should 
be able to answer items 13,14, 20, 7,11, 5,1, 4, 6 and 8; but they 
would have difficulty with items 12,16,10, 3,18,12,19,15,17 and 9. 
Consideration of Figure 3 shows immediately that the items in the 
assessment do not tap the full range of citizenship literacy. Broadly 
speaking, there are many persons in the sample who were well above 
the difficulty level of even the hardest questions. 
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Figure 3: Item-map showing ability and item locations for 
citizenship 
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Discussion and conclusions 

It is impossible to assert unequivocally that this assessment 
confirms that someone possesses a basic knowledge of the English 
language and has an adequate knowledge of Australia and of the 
responsibilities and privileges of Australian citizenship. At best it is a 
partial indicator. 

What can be said, however, is that a substantial proportion of this 
sample would not have passed the cut-off point of 60 per cent (12 
correct out of 20). Around 22 per cent scored 11 or lower, and if this 
were to be increased to 15, then the failure rate would in all likelihood 
be double (approximately 44% in this sample scored 14 or lower). The 
effect is evident in Figure 4 which shows the cumulative proportions 
at each score level. Theses failure rates are far higher than those 
reported in the official statistics where 96.7 per cent passed the test 
on their first or subsequent attempt. The reasons for this difference 
are not clear. 



Score 

Figure 4: Cumulative proportion at each score level and cut-off 
points for passing (60% and 75%) 

A key issue is what constitutes citizenship in this assessment. This is 
shown in the structure or ordering of the items in terms of difficulty in 
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Figure 3. To some this arrangement may seem a bizarre answer, but 
it does represent the empirical difficulty of acquiring these concepts 
in the community. At present it constitutes the best operational 
definition of citizenship literacy that we have available. We could then 
say that an adult who had citizenship literacy in all probability knew 
that the Governor-General was Australia’s head of state, that the first 
European settlers arrived in 1788, that Australian soldiers fought in 
World War I and World War II, that the Judaeo-Christian tradition is 
the source of Australia’s values, that there are six states in Australia, 
that Indigenous people have lived in Australia for at least 40,000 
years, that Edmund Barton was the first Prime Minister and that 
the wattle is Australia’s national flower. Some would argue that this 
does not represent citizenship at all, whereas others might say that 
this is only one component of the construct and reflects the expected 
information acquired through socio-cultural experiences. It is not 
possible to settle such an issue in this paper. 

It is possible, however, to say that many adult Australians have 
performed poorly on this assessment of socio-cultural knowledge, 
irrespective of their citizenship status. This is despite the fact that it is 
a multiple-choice format and only requires recognition and not recall. 
Paradoxically, there are deficiencies in the assessment, especially 
in relation to the fact that the upper levels of citizenship literacy 
are not being assessed. Notwithstanding this limitation, there are 
psychometric merits in this approach, even with such a blunt 20-item 
test, but further refinement of the concept of citizenship is required. 
In addition, the development of an assessment (in whatever form it 
might ultimately take) will help to give an operational meaning to 
citizenship literacy. The educational issue now is that a substantial 
proportion of Australians do not even have the token knowledge that 
is mandated for citizenship or the linguistic competence to read and 
decode the questions. 
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Appendix A 

1. Which colours are represented on the Australian flag? 

a. Green and yellow 

b. Red, black and yellow 

c. Blue, red and white 

d. Orange and purple 

2. Indigenous people have lived in Australia for ... 

a. At least 40,000 years 

b. About 8000 years 

c. About 800 years 

d. Less that 400 years 

3. Australia’s national flower is the ... 

a. Rose 

b. Wattle 

c. Kangaroo paw 

d. Banksia 

4. Which is a popular sport in Australia? 

a. Ice hockey 

b. Water polo 

c. Cricket 

d. Table tennis 
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5. Australia’s political system is a ... 

a. Parliamentary democracy 

b. Monarchy 

c. Dictatorship 

d. Socialist state 

6. The Capital of Australia is... 

a. Sydney 

b. Melbourne 

c. Hobart 

d. Canberra 

7. Which animals are on the Australian Coat of Arms? 

a. Wombat and echidna 

b. Kangaroo and emu 

c. Kangaroo and dingo 

d. Lion and unicorn 

8. Where did the first European settlers to Australia come from? 

a. Spain 

b. France 

c. England 

d. Ireland 

9. Who is Australia’s head of state? 

a. Prime Minister John Howard 

b. Queen Elizabeth II 

c. Governor General Michael Jeffery 

d. Premier Steve Bracks 

10. Who was the first Prime Minister of Australia? 

a. Sir Edmund Barton 

b. Sir Henry Parkes 

c. John Curtin 

d. Sir Robert Menzies 

11. What song is Australia’s national anthem? 

a. God Save the Queen 

b. Star Spangled Banner 

c. Advance Australia Fair 

d. Waltzing Matilda 

12. What do you call the elected head of a state government? 

a. Governor 

b. Premier 

c. Mayor 

d. Prime Minister 

13. Which federal political party or parties are in power? 

a. Australian Labor Party 

b. Australian Democrats and the Australian Greens 

c. National Party 

d. Liberal Party and National Party 
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14. Which of the following are Australian values? 

a. Men and women are equal 

b. 'A fair go’ 

c. Mateship 

d. All of the above 

15. Australia’s values are based on the ... 

a. Teachings of the Koran 

b. The Judaeo-Christian tradition 

c. Catholicism 

d. Secularism 

16. What does Anzac Day commemorate? 

a. The Gallipoli landing 

b. Armistice Day 

c. The Battle of the Somme 

d. Victory in the Pacific 

17. In what year did the first European settlers arrive? 

a. 1801 

b. 1770 

c. 1788 

d. 1505 

18. How many states are there in Australia? 

a. 5 

b. 6 

c. 7 

d. 8 

19. Australian soldiers fought in ... 

a. World War I and World War II 

b. Korean War 

c. Vietnam War 

d. All of the above 

20. What is Australia’s biggest river system? 

a. The Murray Darling 

b. The Murrumbidgee 
c The Yarra 

d. The Mississippi 

Answers: 

1) C, 2) A, 3) B, 4) C, 5) A, 6) D, 7) B, 8) C, 9) B, 10) A, 11) C, 12) B, 13) D, 14) D, 15) B, 
16) A, 17) C, 18) B, 19) D, 20) A 

Source: Herald Sun, May 18, 2007, http://www.news.com.au/heraldsun/ 
story/0,21985,21751638-662,oo.html [retrieved August 2009]. 



